feat: add CacheReadInputTokens and CacheWriteInputTokens to TokenUsageRecord#229
feat: add CacheReadInputTokens and CacheWriteInputTokens to TokenUsageRecord#229
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
pawbana
force-pushed
the
pb/read-write-tokens
branch
2 times, most recently
from
eac105d to
ae135f8
Compare
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
pawbana
force-pushed
the
pb/read-write-tokens
branch
from
7c4899b to
2edd612
Compare
pawbana
force-pushed
the
pb/read-write-tokens
branch
from
2edd612 to
5fb3414
Compare
pawbana
force-pushed
the
pb/read-write-tokens
branch
from
5fb3414 to
6480378
Compare
ssncferreira
left a comment
ssncferreira
left a comment
There was a problem hiding this comment.
LGTM, just a few questions regarding the corresponding coder changes 🦷
| MsgID: completion.ID, | ||
| Input: calculateActualInputTokenUsage(lastUsage), | ||
| Output: lastUsage.CompletionTokens, | ||
| CacheReadInputTokens: lastUsage.PromptTokensDetails.CachedTokens, |
There was a problem hiding this comment.
No CacheWriteInputTokens?
There was a problem hiding this comment.
OpenAI doesn't provide this information.
| "cache_creation_input": resp.Usage.CacheCreationInputTokens, | ||
| "cache_read_input": resp.Usage.CacheReadInputTokens, |
There was a problem hiding this comment.
I'm guessing this will have a corresponding PR on coder side, right? Because IIRC this is being used in the session feature to sum up the tokens. cc @dannykopping
| require.Equal(t, 11904.0, promtest.ToFloat64(m.TokenUseCount.WithLabelValues(config.ProviderOpenAI, "gpt-4.1", "input_cached", defaultActorID, clientLabel))) | ||
| require.Equal(t, 0.0, promtest.ToFloat64(m.TokenUseCount.WithLabelValues(config.ProviderOpenAI, "gpt-4.1", "output_reasoning", defaultActorID, clientLabel))) | ||
| require.Equal(t, 12077.0, promtest.ToFloat64(m.TokenUseCount.WithLabelValues(config.ProviderOpenAI, "gpt-4.1", "total_tokens", defaultActorID, clientLabel))) |
There was a problem hiding this comment.
Do we have any grafana dashboards that use these labels?
There was a problem hiding this comment.
Good question. I've summarized metric labels regarding cached tokens similar to how it is done in DB records but maybe I should keep both.
My intuition was to keep parity between DB records and metric records but it is not possible to update metrics like DB records can be migrated.
But if dashboard are not used much maybe it would be better to just change labels and move fast? I'll check if those labels are used now or not.
2 participants
Added cache read and cache write input token as a first-class value.
Added
CacheReadInputTokensandCacheWriteInputTokensfields toTokenUsageRecordstruct.Removed those values from
ExtraTokenTypesfield.Part 1 of: #150